{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Convolutional Neural Network Example\n",
    "\n",
    "Build a convolutional neural network with TensorFlow v2.\n",
    "\n",
    "This example is using a low-level approach to better understand all mechanics behind building convolutional neural networks and the training process.\n",
    "\n",
    "- Author: Aymeric Damien\n",
    "- Project: https://github.com/aymericdamien/TensorFlow-Examples/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CNN Overview\n",
    "\n",
    "![CNN](http://personal.ie.cuhk.edu.hk/~ccloy/project_target_code/images/fig3.png)\n",
    "\n",
    "## MNIST Dataset Overview\n",
    "\n",
    "This example is using MNIST handwritten digits. The dataset contains 60,000 examples for training and 10,000 examples for testing. The digits have been size-normalized and centered in a fixed-size image (28x28 pixels) with values from 0 to 255. \n",
    "\n",
    "In this example, each image will be converted to float32 and normalized to [0, 1].\n",
    "\n",
    "![MNIST Dataset](http://neuralnetworksanddeeplearning.com/images/mnist_100_digits.png)\n",
    "\n",
    "More info: http://yann.lecun.com/exdb/mnist/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import absolute_import, division, print_function\n",
    "\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras import Model, layers\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copyright 2015 The TensorFlow Authors. All Rights Reserved.\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     http://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License.\n",
    "# ==============================================================================\n",
    "\"\"\"AdaBelief for TensorFlow.\n",
    "Modified from tensorflow/tensorflow/python/training/adam.py\n",
    "\"\"\"\n",
    "\n",
    "from __future__ import absolute_import\n",
    "from __future__ import division\n",
    "from __future__ import print_function\n",
    "\n",
    "from tensorflow.python.eager import context\n",
    "from tensorflow.python.framework import ops\n",
    "from tensorflow.python.ops import control_flow_ops\n",
    "from tensorflow.python.ops import math_ops\n",
    "from tensorflow.python.ops import resource_variable_ops\n",
    "from tensorflow.python.ops import state_ops\n",
    "from tensorflow.python.training import optimizer\n",
    "from tensorflow.python.training import training_ops\n",
    "from tensorflow.python.util.tf_export import tf_export\n",
    "\n",
    "\n",
    "@tf_export(v1=[\"train.AdaBeliefOptimizer\"])\n",
    "class AdaBeliefOptimizer(optimizer.Optimizer):\n",
    "    \"\"\"Optimizer that implements the Adam algorithm.\n",
    "    References:\n",
    "    Adam - A Method for Stochastic Optimization:\n",
    "      [Kingma et al., 2015](https://arxiv.org/abs/1412.6980)\n",
    "      ([pdf](https://arxiv.org/pdf/1412.6980.pdf))\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self,\n",
    "               learning_rate=0.001,\n",
    "               beta1=0.9,\n",
    "               beta2=0.999,\n",
    "               epsilon=1e-8,\n",
    "               use_locking=False,\n",
    "               name=\"AdaBelief\", amsgrad = False):\n",
    "        r\"\"\"Construct a new Adam optimizer.\n",
    "        Initialization:\n",
    "        $$m_0 := 0 \\text{(Initialize initial 1st moment vector)}$$\n",
    "        $$v_0 := 0 \\text{(Initialize initial 2nd moment vector)}$$\n",
    "        $$t := 0 \\text{(Initialize timestep)}$$\n",
    "        The update rule for `variable` with gradient `g` uses an optimization\n",
    "        described at the end of section 2 of the paper:\n",
    "        $$t := t + 1$$\n",
    "        $$\\text{lr}_t := \\mathrm{learning_rate} *\n",
    "          \\sqrt{1 - \\beta_2^t} / (1 - \\beta_1^t)$$\n",
    "        $$m_t := \\beta_1 * m_{t-1} + (1 - \\beta_1) * g$$\n",
    "        $$v_t := \\beta_2 * v_{t-1} + (1 - \\beta_2) * g * g$$\n",
    "        $$\\text{variable} := \\text{variable} -\n",
    "          \\text{lr}_t * m_t / (\\sqrt{v_t} + \\epsilon)$$\n",
    "        The default value of 1e-8 for epsilon might not be a good default in\n",
    "        general. For example, when training an Inception network on ImageNet a\n",
    "        current good choice is 1.0 or 0.1. Note that since AdamOptimizer uses the\n",
    "        formulation just before Section 2.1 of the Kingma and Ba paper rather than\n",
    "        the formulation in Algorithm 1, the \"epsilon\" referred to here is \"epsilon\n",
    "        hat\" in the paper.\n",
    "        The sparse implementation of this algorithm (used when the gradient is an\n",
    "        IndexedSlices object, typically because of `tf.gather` or an embedding\n",
    "        lookup in the forward pass) does apply momentum to variable slices even if\n",
    "        they were not used in the forward pass (meaning they have a gradient equal\n",
    "        to zero). Momentum decay (beta1) is also applied to the entire momentum\n",
    "        accumulator. This means that the sparse behavior is equivalent to the dense\n",
    "        behavior (in contrast to some momentum implementations which ignore momentum\n",
    "        unless a variable slice was actually used).\n",
    "        Args:\n",
    "          learning_rate: A Tensor or a floating point value.  The learning rate.\n",
    "          beta1: A float value or a constant float tensor. The exponential decay\n",
    "            rate for the 1st moment estimates.\n",
    "          beta2: A float value or a constant float tensor. The exponential decay\n",
    "            rate for the 2nd moment estimates.\n",
    "          epsilon: A small constant for numerical stability. This epsilon is\n",
    "            \"epsilon hat\" in the Kingma and Ba paper (in the formula just before\n",
    "            Section 2.1), not the epsilon in Algorithm 1 of the paper.\n",
    "          use_locking: If True use locks for update operations.\n",
    "          name: Optional name for the operations created when applying gradients.\n",
    "            Defaults to \"Adam\".\n",
    "        @compatibility(eager)\n",
    "        When eager execution is enabled, `learning_rate`, `beta1`, `beta2`, and\n",
    "        `epsilon` can each be a callable that takes no arguments and returns the\n",
    "        actual value to use. This can be useful for changing these values across\n",
    "        different invocations of optimizer functions.\n",
    "        @end_compatibility\n",
    "        \"\"\"\n",
    "        super(AdaBeliefOptimizer, self).__init__(use_locking, name)\n",
    "        self._lr = learning_rate\n",
    "        self._beta1 = beta1\n",
    "        self._beta2 = beta2\n",
    "        self._epsilon = epsilon\n",
    "        self.amsgrad = amsgrad\n",
    "\n",
    "        # Tensor versions of the constructor arguments, created in _prepare().\n",
    "        self._lr_t = None\n",
    "        self._beta1_t = None\n",
    "        self._beta2_t = None\n",
    "        self._epsilon_t = None\n",
    "\n",
    "    def _get_beta_accumulators(self):\n",
    "        with ops.init_scope():\n",
    "          if context.executing_eagerly():\n",
    "            graph = None\n",
    "          else:\n",
    "            graph = ops.get_default_graph()\n",
    "          return (self._get_non_slot_variable(\"beta1_power\", graph=graph),\n",
    "                  self._get_non_slot_variable(\"beta2_power\", graph=graph))\n",
    "\n",
    "    def _create_slots(self, var_list):\n",
    "        # Create the beta1 and beta2 accumulators on the same device as the first\n",
    "        # variable. Sort the var_list to make sure this device is consistent across\n",
    "        # workers (these need to go on the same PS, otherwise some updates are\n",
    "        # silently ignored).\n",
    "        first_var = min(var_list, key=lambda x: x.name)\n",
    "        self._create_non_slot_variable(\n",
    "            initial_value=self._beta1, name=\"beta1_power\", colocate_with=first_var)\n",
    "        self._create_non_slot_variable(\n",
    "            initial_value=self._beta2, name=\"beta2_power\", colocate_with=first_var)\n",
    "\n",
    "        # Create slots for the first and second moments.\n",
    "        for v in var_list:\n",
    "          self._zeros_slot(v, \"m\", self._name)\n",
    "          self._zeros_slot(v, \"v\", self._name)\n",
    "          self._zeros_slot(v, \"vhat\", self._name)\n",
    "\n",
    "    def _prepare(self):\n",
    "        lr = self._call_if_callable(self._lr)\n",
    "        beta1 = self._call_if_callable(self._beta1)\n",
    "        beta2 = self._call_if_callable(self._beta2)\n",
    "        epsilon = self._call_if_callable(self._epsilon)\n",
    "\n",
    "        self._lr_t = ops.convert_to_tensor(lr, name=\"learning_rate\")\n",
    "        self._beta1_t = ops.convert_to_tensor(beta1, name=\"beta1\")\n",
    "        self._beta2_t = ops.convert_to_tensor(beta2, name=\"beta2\")\n",
    "        self._epsilon_t = ops.convert_to_tensor(epsilon, name=\"epsilon\")\n",
    "\n",
    "    def _apply_dense(self, grad, var):\n",
    "          graph = None if context.executing_eagerly() else ops.get_default_graph()\n",
    "          beta1_power = math_ops.cast(self._get_non_slot_variable(\"beta1_power\", graph=graph), var.dtype.base_dtype)\n",
    "          beta2_power = math_ops.cast(self._get_non_slot_variable(\"beta2_power\", graph=graph), var.dtype.base_dtype)\n",
    "          lr_t = math_ops.cast(self._lr_t, var.dtype.base_dtype)\n",
    "\n",
    "          beta1_t = math_ops.cast(self._beta1_t, var.dtype.base_dtype)\n",
    "          beta2_t = math_ops.cast(self._beta2_t, var.dtype.base_dtype)\n",
    "          epsilon_t = math_ops.cast(self._epsilon_t, var.dtype.base_dtype)\n",
    "\n",
    "          step_size = (lr_t * math_ops.sqrt(1 - beta2_power) / (1 - beta1_power))\n",
    "\n",
    "          # m_t = beta1 * m + (1 - beta1) * g_t\n",
    "          m = self.get_slot(var, \"m\")\n",
    "          m_scaled_g_values = grad * (1 - beta1_t)\n",
    "          m_t = state_ops.assign(m, beta1_t * m + m_scaled_g_values, use_locking=self._use_locking)\n",
    "\n",
    "          # v_t = beta2 * v + (1 - beta2) * (g_t * g_t)\n",
    "          v = self.get_slot(var, \"v\")\n",
    "          v_scaled_g_values = (grad -m_t) * (grad - m_t) * (1 - beta2_t)\n",
    "          v_t = state_ops.assign(v, beta2_t * v + v_scaled_g_values + epsilon_t, use_locking=self._use_locking)\n",
    "\n",
    "          # amsgrad\n",
    "          vhat = self.get_slot(var, \"vhat\")\n",
    "          if self.amsgrad:\n",
    "              vhat_t = state_ops.assign(vhat, math_ops.maximum(v_t, vhat))\n",
    "              v_sqrt = math_ops.sqrt(vhat_t)\n",
    "          else:\n",
    "              vhat_t = state_ops.assign(vhat, vhat)\n",
    "              v_sqrt = math_ops.sqrt(v_t)\n",
    "\n",
    "          # Compute the bounds\n",
    "          step_size = step_size / (v_sqrt + epsilon_t)\n",
    "          bounded_lr = m_t * step_size\n",
    "\n",
    "          var_update = state_ops.assign_sub(var, bounded_lr, use_locking=self._use_locking)\n",
    "          return control_flow_ops.group(*[var_update, m_t, v_t, vhat_t])\n",
    "\n",
    "    def _resource_apply_dense(self, grad, var):\n",
    "          graph = None if context.executing_eagerly() else ops.get_default_graph()\n",
    "          beta1_power = math_ops.cast(self._get_non_slot_variable(\"beta1_power\", graph=graph), grad.dtype.base_dtype)\n",
    "          beta2_power = math_ops.cast(self._get_non_slot_variable(\"beta2_power\", graph=graph), grad.dtype.base_dtype)\n",
    "          lr_t = math_ops.cast(self._lr_t, grad.dtype.base_dtype)\n",
    "          beta1_t = math_ops.cast(self._beta1_t, grad.dtype.base_dtype)\n",
    "          beta2_t = math_ops.cast(self._beta2_t, grad.dtype.base_dtype)\n",
    "          epsilon_t = math_ops.cast(self._epsilon_t, grad.dtype.base_dtype)\n",
    "\n",
    "          step_size = (lr_t * math_ops.sqrt(1 - beta2_power) / (1 - beta1_power))\n",
    "\n",
    "          # m_t = beta1 * m + (1 - beta1) * g_t\n",
    "          m = self.get_slot(var, \"m\")\n",
    "          m_scaled_g_values = grad * (1 - beta1_t)\n",
    "          m_t = state_ops.assign(m, beta1_t * m + m_scaled_g_values, use_locking=self._use_locking)\n",
    "\n",
    "          # v_t = beta2 * v + (1 - beta2) * (g_t * g_t)\n",
    "          v = self.get_slot(var, \"v\")\n",
    "          v_scaled_g_values = (grad - m_t) * (grad - m_t) * (1 - beta2_t)\n",
    "          v_t = state_ops.assign(v, beta2_t * v + v_scaled_g_values + epsilon_t, use_locking=self._use_locking)\n",
    "\n",
    "          # amsgrad\n",
    "          vhat = self.get_slot(var, \"vhat\")\n",
    "          if self.amsgrad:\n",
    "              vhat_t = state_ops.assign(vhat, math_ops.maximum(v_t, vhat))\n",
    "              v_sqrt = math_ops.sqrt(vhat_t)\n",
    "          else:\n",
    "              vhat_t = state_ops.assign(vhat, vhat)\n",
    "              v_sqrt = math_ops.sqrt(v_t)\n",
    "\n",
    "          # Compute the bounds\n",
    "          step_size = step_size / (v_sqrt + epsilon_t)\n",
    "          bounded_lr = m_t * step_size\n",
    "\n",
    "          var_update = state_ops.assign_sub(var, bounded_lr, use_locking=self._use_locking)\n",
    "\n",
    "          return control_flow_ops.group(*[var_update, m_t, v_t, vhat_t])\n",
    "\n",
    "    def _apply_sparse_shared(self, grad, var, indices, scatter_add):\n",
    "        beta1_power, beta2_power = self._get_beta_accumulators()\n",
    "        beta1_power = math_ops.cast(beta1_power, var.dtype.base_dtype)\n",
    "        beta2_power = math_ops.cast(beta2_power, var.dtype.base_dtype)\n",
    "        lr_t = math_ops.cast(self._lr_t, var.dtype.base_dtype)\n",
    "        beta1_t = math_ops.cast(self._beta1_t, var.dtype.base_dtype)\n",
    "        beta2_t = math_ops.cast(self._beta2_t, var.dtype.base_dtype)\n",
    "        epsilon_t = math_ops.cast(self._epsilon_t, var.dtype.base_dtype)\n",
    "        lr = (lr_t * math_ops.sqrt(1 - beta2_power) / (1 - beta1_power))\n",
    "        # m_t = beta1 * m + (1 - beta1) * g_t\n",
    "        m = self.get_slot(var, \"m\")\n",
    "        m_scaled_g_values = grad * (1 - beta1_t)\n",
    "        m_t = state_ops.assign(m, m * beta1_t, use_locking=self._use_locking)\n",
    "        with ops.control_dependencies([m_t]):\n",
    "            m_t = scatter_add(m, indices, m_scaled_g_values)\n",
    "        # v_t = beta2 * v + (1 - beta2) * (g_t * g_t)\n",
    "        v = self.get_slot(var, \"v\")\n",
    "        v_scaled_g_values = (grad - m_t) * (grad - m_t) * (1 - beta2_t)\n",
    "        v_t = state_ops.assign(v, v * beta2_t, use_locking=self._use_locking)\n",
    "        with ops.control_dependencies([v_t]):\n",
    "            v_t = scatter_add(v, indices, v_scaled_g_values + epsilon_t)\n",
    "\n",
    "        # amsgrad\n",
    "        vhat = self.get_slot(var, \"vhat\")\n",
    "        if self.amsgrad:\n",
    "            vhat_t = state_ops.assign(vhat, math_ops.maximum(v_t, vhat))\n",
    "            v_sqrt = math_ops.sqrt(vhat_t)\n",
    "        else:\n",
    "            vhat_t = state_ops.assign(vhat, vhat)\n",
    "            v_sqrt = math_ops.sqrt(v_t)\n",
    "\n",
    "        var_update = state_ops.assign_sub(\n",
    "            var, lr * m_t / (v_sqrt + epsilon_t), use_locking=self._use_locking)\n",
    "        return control_flow_ops.group(*[var_update, m_t, v_t, vhat_t])\n",
    "\n",
    "    def _apply_sparse(self, grad, var):\n",
    "        return self._apply_sparse_shared(\n",
    "            grad.values,\n",
    "            var,\n",
    "            grad.indices,\n",
    "            lambda x, i, v: state_ops.scatter_add(  # pylint: disable=g-long-lambda\n",
    "                x,\n",
    "                i,\n",
    "                v,\n",
    "                use_locking=self._use_locking))\n",
    "\n",
    "    def _resource_scatter_add(self, x, i, v):\n",
    "        with ops.control_dependencies(\n",
    "            [resource_variable_ops.resource_scatter_add(x.handle, i, v)]):\n",
    "            return x.value()\n",
    "\n",
    "    def _resource_apply_sparse(self, grad, var, indices):\n",
    "        return self._apply_sparse_shared(grad, var, indices,\n",
    "                                         self._resource_scatter_add)\n",
    "\n",
    "    def _finish(self, update_ops, name_scope):\n",
    "        # Update the power accumulators.\n",
    "        with ops.control_dependencies(update_ops):\n",
    "            beta1_power, beta2_power = self._get_beta_accumulators()\n",
    "            with ops.colocate_with(beta1_power):\n",
    "                update_beta1 = beta1_power.assign(\n",
    "                               beta1_power * self._beta1_t, use_locking=self._use_locking)\n",
    "                update_beta2 = beta2_power.assign(\n",
    "                                beta2_power * self._beta2_t, use_locking=self._use_locking)\n",
    "        return control_flow_ops.group(\n",
    "            *update_ops + [update_beta1, update_beta2], name=name_scope)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "# MNIST dataset parameters.\n",
    "num_classes = 10 # total classes (0-9 digits).\n",
    "\n",
    "# Training parameters.\n",
    "learning_rate = 0.001\n",
    "training_steps = 200\n",
    "batch_size = 128\n",
    "display_step = 10\n",
    "\n",
    "# Network parameters.\n",
    "conv1_filters = 32 # number of filters for 1st conv layer.\n",
    "conv2_filters = 64 # number of filters for 2nd conv layer.\n",
    "fc1_units = 1024 # number of neurons for 1st fully-connected layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prepare MNIST data.\n",
    "from tensorflow.keras.datasets import mnist, cifar10\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
    "# Convert to float32.\n",
    "x_train, x_test = np.array(x_train, np.float32), np.array(x_test, np.float32)\n",
    "# Normalize images value from [0, 255] to [0, 1].\n",
    "x_train, x_test = x_train / 255., x_test / 255."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use tf.data API to shuffle and batch data.\n",
    "train_data = tf.data.Dataset.from_tensor_slices((x_train, y_train))\n",
    "train_data = train_data.repeat().shuffle(5000).batch(batch_size).prefetch(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create TF Model.\n",
    "class ConvNet(Model):\n",
    "    # Set layers.\n",
    "    def __init__(self):\n",
    "        super(ConvNet, self).__init__()\n",
    "        # Convolution Layer with 32 filters and a kernel size of 5.\n",
    "        self.conv1 = layers.Conv2D(32, kernel_size=5, activation=tf.nn.relu)\n",
    "        # Max Pooling (down-sampling) with kernel size of 2 and strides of 2. \n",
    "        self.maxpool1 = layers.MaxPool2D(2, strides=2)\n",
    "\n",
    "        # Convolution Layer with 64 filters and a kernel size of 3.\n",
    "        self.conv2 = layers.Conv2D(64, kernel_size=3, activation=tf.nn.relu)\n",
    "        # Max Pooling (down-sampling) with kernel size of 2 and strides of 2. \n",
    "        self.maxpool2 = layers.MaxPool2D(2, strides=2)\n",
    "\n",
    "        # Flatten the data to a 1-D vector for the fully connected layer.\n",
    "        self.flatten = layers.Flatten()\n",
    "\n",
    "        # Fully connected layer.\n",
    "        self.fc1 = layers.Dense(1024)\n",
    "        # Apply Dropout (if is_training is False, dropout is not applied).\n",
    "        self.dropout = layers.Dropout(rate=0.5)\n",
    "\n",
    "        # Output layer, class prediction.\n",
    "        self.out = layers.Dense(num_classes)\n",
    "\n",
    "    # Set forward pass.\n",
    "    def call(self, x, is_training=False):\n",
    "        x = tf.reshape(x, [-1, 28, 28, 1])\n",
    "        x = self.conv1(x)\n",
    "        x = self.maxpool1(x)\n",
    "        x = self.conv2(x)\n",
    "        x = self.maxpool2(x)\n",
    "        x = self.flatten(x)\n",
    "        x = self.fc1(x)\n",
    "        x = self.dropout(x, training=is_training)\n",
    "        x = self.out(x)\n",
    "        if not is_training:\n",
    "            # tf cross entropy expect logits without softmax, so only\n",
    "            # apply softmax when not training.\n",
    "            x = tf.nn.softmax(x)\n",
    "        return x\n",
    "\n",
    "# Build neural network model.\n",
    "conv_net = ConvNet()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cross-Entropy Loss.\n",
    "# Note that this will apply 'softmax' to the logits.\n",
    "def cross_entropy_loss(x, y):\n",
    "    # Convert labels to int 64 for tf cross-entropy function.\n",
    "    y = tf.cast(y, tf.int64)\n",
    "    # Apply softmax to logits and compute cross-entropy.\n",
    "    loss = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y, logits=x)\n",
    "    # Average loss across the batch.\n",
    "    return tf.reduce_mean(loss)\n",
    "\n",
    "# Accuracy metric.\n",
    "def accuracy(y_pred, y_true):\n",
    "    # Predicted class is the index of highest score in prediction vector (i.e. argmax).\n",
    "    correct_prediction = tf.equal(tf.argmax(y_pred, 1), tf.cast(y_true, tf.int64))\n",
    "    return tf.reduce_mean(tf.cast(correct_prediction, tf.float32), axis=-1)\n",
    "\n",
    "# Stochastic gradient descent optimizer.\n",
    "optimizer = AdaBeliefOptimizer(learning_rate, epsilon=1e-14) # Deafault epsilon for Adam is 1e-7 in tensorlfow, 1e-8 in pytorch\n",
    "#optimizer = tf.optimizers.Adam(learning_rate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Optimization process. \n",
    "def run_optimization(x, y):\n",
    "    # Wrap computation inside a GradientTape for automatic differentiation.\n",
    "    with tf.GradientTape() as g:\n",
    "        # Forward pass.\n",
    "        pred = conv_net(x, is_training=True)\n",
    "        # Compute loss.\n",
    "        loss = cross_entropy_loss(pred, y)\n",
    "        \n",
    "    # Variables to update, i.e. trainable variables.\n",
    "    trainable_variables = conv_net.trainable_variables\n",
    "\n",
    "    # Compute gradients.\n",
    "    gradients = g.gradient(loss, trainable_variables)\n",
    "    \n",
    "    # Update W and b following gradients.\n",
    "    optimizer.apply_gradients(zip(gradients, trainable_variables))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step: 10, loss: 1.802794, accuracy: 0.820312\n",
      "step: 20, loss: 1.639538, accuracy: 0.890625\n",
      "step: 30, loss: 1.539099, accuracy: 0.976562\n",
      "step: 40, loss: 1.555407, accuracy: 0.945312\n",
      "step: 50, loss: 1.529462, accuracy: 0.968750\n",
      "step: 60, loss: 1.531016, accuracy: 0.945312\n",
      "step: 70, loss: 1.521319, accuracy: 0.960938\n",
      "step: 80, loss: 1.552162, accuracy: 0.929688\n",
      "step: 90, loss: 1.495071, accuracy: 0.984375\n",
      "step: 100, loss: 1.490701, accuracy: 0.984375\n",
      "step: 110, loss: 1.522786, accuracy: 0.953125\n",
      "step: 120, loss: 1.487235, accuracy: 0.984375\n",
      "step: 130, loss: 1.508773, accuracy: 0.976562\n",
      "step: 140, loss: 1.530836, accuracy: 0.945312\n",
      "step: 150, loss: 1.489567, accuracy: 0.984375\n",
      "step: 160, loss: 1.484169, accuracy: 0.984375\n",
      "step: 170, loss: 1.496965, accuracy: 0.984375\n",
      "step: 180, loss: 1.498039, accuracy: 0.968750\n",
      "step: 190, loss: 1.483729, accuracy: 1.000000\n",
      "step: 200, loss: 1.513226, accuracy: 0.968750\n"
     ]
    }
   ],
   "source": [
    "# Run training for the given number of steps.\n",
    "for step, (batch_x, batch_y) in enumerate(train_data.take(training_steps), 1):\n",
    "    # Run the optimization to update W and b values.\n",
    "    run_optimization(batch_x, batch_y)\n",
    "    \n",
    "    if step % display_step == 0:\n",
    "        pred = conv_net(batch_x)\n",
    "        loss = cross_entropy_loss(pred, batch_y)\n",
    "        acc = accuracy(pred, batch_y)\n",
    "        print(\"step: %i, loss: %f, accuracy: %f\" % (step, loss, acc))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Accuracy: 0.9790999889373779\n"
     ]
    }
   ],
   "source": [
    "# Test model on validation set.\n",
    "pred = conv_net(x_test)\n",
    "print(\"Test Accuracy: {}\" .format( accuracy(pred, y_test)) )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Visualize predictions.\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAM4ElEQVR4nO3db6xU9Z3H8c9nWZoY6QNQce9alC7xgc3GgCIxQTfXkDYsPsBGuikPGjZpvH2Apo0NWeM+wIeN2bZZn5DcRlO6YW1IqEqMcSHYSBq18WJQLr0BkbBwyxVsMCmYGES/++AeN1ecc2acMzNn4Pt+JZOZOd85Z74Z7odz5vyZnyNCAK5+f9N0AwAGg7ADSRB2IAnCDiRB2IEk/naQb2abXf9An0WEW02vtWa3vdb2EdvHbD9WZ1kA+svdHme3PU/SUUnfljQt6U1JGyPiTxXzsGYH+qwfa/ZVko5FxPGIuCjpt5LW11gegD6qE/abJJ2a83y6mPYFtsdsT9ieqPFeAGqqs4Ou1abClzbTI2Jc0rjEZjzQpDpr9mlJS+Y8/4ak0/XaAdAvdcL+pqRbbX/T9tckfV/S7t60BaDXut6Mj4hLth+W9D+S5kl6JiIO96wzAD3V9aG3rt6M7+xA3/XlpBoAVw7CDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBJdj88uSbZPSDov6VNJlyJiZS+aAtB7tcJeuC8i/tKD5QDoIzbjgSTqhj0k7bF9wPZYqxfYHrM9YXui5nsBqMER0f3M9t9HxGnbiyXtlfRIROyveH33bwagIxHhVtNrrdkj4nRxf1bSc5JW1VkegP7pOuy2r7X99c8fS/qOpMleNQagt+rsjb9R0nO2P1/Of0fEyz3pCkDP1frO/pXfjO/sQN/15Ts7gCsHYQeSIOxAEoQdSIKwA0n04kKYFDZs2FBae+ihhyrnPX36dGX9448/rqzv2LGjsv7++++X1o4dO1Y5L/JgzQ4kQdiBJAg7kARhB5Ig7EAShB1IgrADSXDVW4eOHz9eWlu6dOngGmnh/PnzpbXDhw8PsJPhMj09XVp78sknK+edmLhyf0WNq96A5Ag7kARhB5Ig7EAShB1IgrADSRB2IAmuZ+9Q1TXrt99+e+W8U1NTlfXbbrutsn7HHXdU1kdHR0trd999d+W8p06dqqwvWbKksl7HpUuXKusffPBBZX1kZKTr9z558mRl/Uo+zl6GNTuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJMH17FeBhQsXltaWL19eOe+BAwcq63fddVc3LXWk3e/lHz16tLLe7vyFRYsWldY2b95cOe+2bdsq68Os6+vZbT9j+6ztyTnTFtnea/vd4r78rw3AUOhkM/7XktZeNu0xSfsi4lZJ+4rnAIZY27BHxH5J5y6bvF7S9uLxdkkP9LYtAL3W7bnxN0bEjCRFxIztxWUvtD0maazL9wHQI32/ECYixiWNS+ygA5rU7aG3M7ZHJKm4P9u7lgD0Q7dh3y1pU/F4k6QXetMOgH5pe5zd9rOSRiVdL+mMpK2Snpe0U9LNkk5K+l5EXL4Tr9Wy2IxHxx588MHK+s6dOyvrk5OTpbX77ruvct5z59r+OQ+tsuPsbb+zR8TGktKaWh0BGChOlwWSIOxAEoQdSIKwA0kQdiAJLnFFYxYvLj3LWpJ06NChWvNv2LChtLZr167Kea9kDNkMJEfYgSQIO5AEYQeSIOxAEoQdSIKwA0kwZDMa0+7nnG+44YbK+ocfflhZP3LkyFfu6WrGmh1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkuB6dvTV6tWrS2uvvPJK5bzz58+vrI+OjlbW9+/fX1m/WnE9O5AcYQeSIOxAEoQdSIKwA0kQdiAJwg4kwfXs6Kt169aV1todR9+3b19l/fXXX++qp6zartltP2P7rO3JOdOesP1n2weLW/m/KICh0Mlm/K8lrW0x/ZcRsby4vdTbtgD0WtuwR8R+SecG0AuAPqqzg+5h2+8Um/kLy15ke8z2hO2JGu8FoKZuw75N0jJJyyXNSPp52QsjYjwiVkbEyi7fC0APdBX2iDgTEZ9GxGeSfiVpVW/bAtBrXYXd9sicp9+VNFn2WgDDoe1xdtvPShqVdL3taUlbJY3aXi4pJJ2Q9KP+tYhhds0111TW165tdSBn1sWLFyvn3bp1a2X9k08+qazji9qGPSI2tpj8dB96AdBHnC4LJEHYgSQIO5AEYQeSIOxAElziilq2bNlSWV+xYkVp7eWXX66c97XXXuuqJ7TGmh1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkmDIZlS6//77K+vPP/98Zf2jjz4qrVVd/ipJb7zxRmUdrTFkM5AcYQeSIOxAEoQdSIKwA0kQdiAJwg4kwfXsyV133XWV9aeeeqqyPm/evMr6Sy+Vj/nJcfTBYs0OJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0lwPftVrt1x8HbHuu+8887K+nvvvVdZr7pmvd286E7X17PbXmL797anbB+2/eNi+iLbe22/W9wv7HXTAHqnk834S5J+GhG3Sbpb0mbb35L0mKR9EXGrpH3FcwBDqm3YI2ImIt4qHp+XNCXpJknrJW0vXrZd0gN96hFAD3ylc+NtL5W0QtIfJd0YETPS7H8ItheXzDMmaaxmnwBq6jjsthdI2iXpJxHxV7vlPoAviYhxSePFMthBBzSko0NvtudrNug7IuJ3xeQztkeK+oiks/1pEUAvtF2ze3YV/rSkqYj4xZzSbkmbJP2suH+hLx2ilmXLllXW2x1aa+fRRx+trHN4bXh0shm/WtIPJB2yfbCY9rhmQ77T9g8lnZT0vb50CKAn2oY9Iv4gqewL+pretgOgXzhdFkiCsANJEHYgCcIOJEHYgST4KemrwC233FJa27NnT61lb9mypbL+4osv1lo+Boc1O5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kwXH2q8DYWPmvft188821lv3qq69W1gf5U+SohzU7kARhB5Ig7EAShB1IgrADSRB2IAnCDiTBcfYrwD333FNZf+SRRwbUCa5krNmBJAg7kARhB5Ig7EAShB1IgrADSRB2IIlOxmdfIuk3kv5O0meSxiPiP20/IekhSR8UL308Il7qV6OZ3XvvvZX1BQsWdL3sduOnX7hwoetlY7h0clLNJUk/jYi3bH9d0gHbe4vaLyPiP/rXHoBe6WR89hlJM8Xj87anJN3U78YA9NZX+s5ue6mkFZL+WEx62PY7tp+xvbBknjHbE7Yn6rUKoI6Ow257gaRdkn4SEX+VtE3SMknLNbvm/3mr+SJiPCJWRsTK+u0C6FZHYbc9X7NB3xERv5OkiDgTEZ9GxGeSfiVpVf/aBFBX27DbtqSnJU1FxC/mTB+Z87LvSprsfXsAeqWTvfGrJf1A0iHbB4tpj0vaaHu5pJB0QtKP+tAfanr77bcr62vWrKmsnzt3rpftoEGd7I3/gyS3KHFMHbiCcAYdkARhB5Ig7EAShB1IgrADSRB2IAkPcshd24zvC/RZRLQ6VM6aHciCsANJEHYgCcIOJEHYgSQIO5AEYQeSGPSQzX+R9L9znl9fTBtGw9rbsPYl0Vu3etnbLWWFgZ5U86U3tyeG9bfphrW3Ye1LorduDao3NuOBJAg7kETTYR9v+P2rDGtvw9qXRG/dGkhvjX5nBzA4Ta/ZAQwIYQeSaCTsttfaPmL7mO3HmuihjO0Ttg/ZPtj0+HTFGHpnbU/OmbbI9l7b7xb3LcfYa6i3J2z/ufjsDtpe11BvS2z/3vaU7cO2f1xMb/Szq+hrIJ/bwL+z254n6aikb0ualvSmpI0R8aeBNlLC9glJKyOi8RMwbP+TpAuSfhMR/1hMe1LSuYj4WfEf5cKI+Lch6e0JSReaHsa7GK1oZO4w45IekPSvavCzq+jrXzSAz62JNfsqScci4nhEXJT0W0nrG+hj6EXEfkmXD8myXtL24vF2zf6xDFxJb0MhImYi4q3i8XlJnw8z3uhnV9HXQDQR9psknZrzfFrDNd57SNpj+4DtsaabaeHGiJiRZv94JC1uuJ/LtR3Ge5AuG2Z8aD67boY/r6uJsLf6faxhOv63OiLukPTPkjYXm6voTEfDeA9Ki2HGh0K3w5/X1UTYpyUtmfP8G5JON9BHSxFxurg/K+k5Dd9Q1Gc+H0G3uD/bcD//b5iG8W41zLiG4LNrcvjzJsL+pqRbbX/T9tckfV/S7gb6+BLb1xY7TmT7Wknf0fANRb1b0qbi8SZJLzTYyxcMyzDeZcOMq+HPrvHhzyNi4DdJ6zS7R/49Sf/eRA8lff2DpLeL2+Gme5P0rGY36z7R7BbRDyVdJ2mfpHeL+0VD1Nt/STok6R3NBmukod7u0exXw3ckHSxu65r+7Cr6GsjnxumyQBKcQQckQdiBJAg7kARhB5Ig7EAShB1IgrADSfwfrLwRQB25h+kAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model prediction: 7\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAANYUlEQVR4nO3df6hc9ZnH8c9n3QTEFk0ihouRtUaF1UWtXGXRsrjURlc0MWDXBFlcVrj9o0LF+CNkhQiLKLvb3T8DtzQ0atemITGNtWwqof5YMMGrxJg0aTUS0zTXXLIBmyBSkzz7xz13uU3unLk5Z2bOJM/7BZeZOc/M9zyMfnLOzJlzvo4IATj3/VnTDQDoDcIOJEHYgSQIO5AEYQeS+PNersw2X/0DXRYRnmp5rS277Ttt/8b2R7aX1xkLQHe56nF22+dJ+q2kb0k6IOkdSUsj4tclr2HLDnRZN7bsN0v6KCI+jog/SvqJpEU1xgPQRXXCfqmk3016fKBY9idsD9kesT1SY10AaqrzBd1Uuwqn7aZHxLCkYYndeKBJdbbsByRdNunxPEkH67UDoFvqhP0dSVfZ/prtmZKWSNrUmbYAdFrl3fiIOG77YUmbJZ0naXVE7OpYZwA6qvKht0or4zM70HVd+VENgLMHYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSfT0UtKo5rHHHiutn3/++S1r1113Xelr77vvvko9TVi1alVp/e23325Ze+GFF2qtG2eGLTuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJMHVZfvA2rVrS+t1j4U3ae/evS1rt99+e+lr9+/f3+l2UuDqskByhB1IgrADSRB2IAnCDiRB2IEkCDuQBOez90CTx9H37NlTWt+8eXNp/Yorriit33PPPaX1+fPnt6w98MADpa999tlnS+s4M7XCbnufpKOSTkg6HhGDnWgKQOd1Ysv+txFxuAPjAOgiPrMDSdQNe0j6pe13bQ9N9QTbQ7ZHbI/UXBeAGuruxt8aEQdtXyLpNdt7IuLNyU+IiGFJwxInwgBNqrVlj4iDxe2YpJcl3dyJpgB0XuWw277A9lcn7ktaIGlnpxoD0Fl1duPnSnrZ9sQ4/xUR/92Rrs4yg4PlRxwXL15ca/xdu3aV1hcuXNiydvhw+YGSY8eOldZnzpxZWt+6dWtp/frrr29ZmzNnTulr0VmVwx4RH0tq/V8SQF/h0BuQBGEHkiDsQBKEHUiCsANJcIprBwwMDJTWi8OTLbU7tHbHHXeU1kdHR0vrdSxbtqy0fs0111Qe+9VXX638Wpw5tuxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kATH2TvglVdeKa1feeWVpfWjR4+W1o8cOXLGPXXKkiVLSuszZszoUSeoiy07kARhB5Ig7EAShB1IgrADSRB2IAnCDiTBcfYe+OSTT5puoaXHH3+8tH711VfXGn/btm2Vaug8tuxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kIQjoncrs3u3MkiS7r777tL6unXrSuvtpmweGxsrrZedD//GG2+UvhbVRMSUExW03bLbXm17zPbOSctm237N9ofF7axONgug86azG/8jSXeesmy5pC0RcZWkLcVjAH2sbdgj4k1Jp14XaZGkNcX9NZLu7WxbADqt6m/j50bEqCRFxKjtS1o90faQpKGK6wHQIV0/ESYihiUNS3xBBzSp6qG3Q7YHJKm4Lf9KFkDjqoZ9k6QHi/sPSvpZZ9oB0C1td+NtvyTpNkkX2z4gaaWk5yT91PZDkvZL+nY3m0R1g4ODpfV2x9HbWbt2bWmdY+n9o23YI2Jpi9I3O9wLgC7i57JAEoQdSIKwA0kQdiAJwg4kwaWkzwEbN25sWVuwYEGtsZ9//vnS+lNPPVVrfPQOW3YgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIJLSZ8FBgYGSuvvv/9+y9qcOXNKX3v48OHS+i233FJa37t3b2kdvVf5UtIAzg2EHUiCsANJEHYgCcIOJEHYgSQIO5AE57OfBdavX19ab3csvcyLL75YWuc4+rmDLTuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJMFx9j6wcOHC0vqNN95YeezXX3+9tL5y5crKY+Ps0nbLbnu17THbOycte9r2721vL/7u6m6bAOqazm78jyTdOcXy/4yIG4q/X3S2LQCd1jbsEfGmpCM96AVAF9X5gu5h2zuK3fxZrZ5ke8j2iO2RGusCUFPVsK+SNF/SDZJGJX2/1RMjYjgiBiNisOK6AHRApbBHxKGIOBERJyX9QNLNnW0LQKdVCrvtydc2XixpZ6vnAugPbY+z235J0m2SLrZ9QNJKSbfZvkFSSNon6Tvda/Hs1+588xUrVpTWZ8yYUXnd27dvL60fO3as8tg4u7QNe0QsnWLxD7vQC4Au4ueyQBKEHUiCsANJEHYgCcIOJMEprj2wbNmy0vpNN91Ua/yNGze2rHEKKyawZQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJBwRvVuZ3buV9ZEvvviitF7nFFZJmjdvXsva6OhorbFx9okIT7WcLTuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJMH57OeA2bNnt6x9+eWXPezkdJ999lnLWrve2v3+4MILL6zUkyRddNFFpfVHH3208tjTceLEiZa1J598svS1n3/+eaV1smUHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQ4zn4O2LFjR9MttLRu3bqWtXbn2s+dO7e0fv/991fqqd99+umnpfVnnnmm0rhtt+y2L7P9K9u7be+y/b1i+Wzbr9n+sLidVakDAD0xnd3445KWRcRfSvprSd+1fY2k5ZK2RMRVkrYUjwH0qbZhj4jRiHivuH9U0m5Jl0paJGlN8bQ1ku7tUo8AOuCMPrPbvlzS1yVtkzQ3Ikal8X8QbF/S4jVDkoZq9gmgpmmH3fZXJK2X9EhE/MGe8pp2p4mIYUnDxRgpLzgJ9INpHXqzPUPjQf9xRGwoFh+yPVDUBySNdadFAJ3Q9lLSHt+Er5F0JCIembT83yT9b0Q8Z3u5pNkR8USbsVJu2Tds2FBaX7RoUY86yeX48eMtaydPnqw19qZNm0rrIyMjlcd+6623Sutbt24trbe6lPR0duNvlfQPkj6wvb1YtkLSc5J+avshSfslfXsaYwFoSNuwR8T/SGr1Af2bnW0HQLfwc1kgCcIOJEHYgSQIO5AEYQeSYMrmPvDEE6U/T6g9pXOZa6+9trTezdNIV69eXVrft29frfHXr1/fsrZnz55aY/czpmwGkiPsQBKEHUiCsANJEHYgCcIOJEHYgSQ4zg6cYzjODiRH2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0m0Dbvty2z/yvZu27tsf69Y/rTt39veXvzd1f12AVTV9uIVtgckDUTEe7a/KuldSfdK+ntJxyLi36e9Mi5eAXRdq4tXTGd+9lFJo8X9o7Z3S7q0s+0B6LYz+sxu+3JJX5e0rVj0sO0dtlfbntXiNUO2R2yP1GsVQB3Tvgad7a9IekPSMxGxwfZcSYclhaR/0fiu/j+1GYPdeKDLWu3GTyvstmdI+rmkzRHxH1PUL5f084j4qzbjEHagyypfcNK2Jf1Q0u7JQS++uJuwWNLOuk0C6J7pfBv/DUlvSfpA0sli8QpJSyXdoPHd+H2SvlN8mVc2Flt2oMtq7cZ3CmEHuo/rxgPJEXYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Joe8HJDjss6ZNJjy8ulvWjfu2tX/uS6K2qTvb2F60KPT2f/bSV2yMRMdhYAyX6tbd+7Uuit6p61Ru78UAShB1IoumwDze8/jL92lu/9iXRW1U96a3Rz+wAeqfpLTuAHiHsQBKNhN32nbZ/Y/sj28ub6KEV2/tsf1BMQ93o/HTFHHpjtndOWjbb9mu2Pyxup5xjr6He+mIa75Jpxht975qe/rznn9ltnyfpt5K+JemApHckLY2IX/e0kRZs75M0GBGN/wDD9t9IOibp+YmptWz/q6QjEfFc8Q/lrIh4sk96e1pnOI13l3prNc34P6rB966T059X0cSW/WZJH0XExxHxR0k/kbSogT76XkS8KenIKYsXSVpT3F+j8f9Zeq5Fb30hIkYj4r3i/lFJE9OMN/relfTVE02E/VJJv5v0+ID6a773kPRL2+/aHmq6mSnMnZhmq7i9pOF+TtV2Gu9eOmWa8b5576pMf15XE2Gfamqafjr+d2tE3Cjp7yR9t9hdxfSskjRf43MAjkr6fpPNFNOMr5f0SET8ocleJpuir568b02E/YCkyyY9nifpYAN9TCkiDha3Y5Je1vjHjn5yaGIG3eJ2rOF+/l9EHIqIExFxUtIP1OB7V0wzvl7SjyNiQ7G48fduqr569b41EfZ3JF1l+2u2Z0paImlTA32cxvYFxRcnsn2BpAXqv6moN0l6sLj/oKSfNdjLn+iXabxbTTOuht+7xqc/j4ie/0m6S+PfyO+V9M9N9NCiryskvV/87Wq6N0kvaXy37kuN7xE9JGmOpC2SPixuZ/dRby9ofGrvHRoP1kBDvX1D4x8Nd0jaXvzd1fR7V9JXT943fi4LJMEv6IAkCDuQBGEHkiDsQBKEHUiCsANJEHYgif8DskwsZgRKJ/QAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model prediction: 2\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAMD0lEQVR4nO3dXagc5R3H8d+vabwwepFUE0OUxIqiRTEpQYSEavEFG4SYC4sRSqTC8cJAhF5U7IVCKUio9sIL4YjBVKwvRINR60sIkrQ3mqOmGo1GK6kec8hRFHxDrMm/F2dSjvHs7HFnZmc9/+8HDrs7z87OnyG/PM/szOzjiBCAme9HbRcAoD8IO5AEYQeSIOxAEoQdSOLH/dyYbb76BxoWEZ5qeaWe3fYVtt+y/Y7tm6t8FoBmudfz7LZnSdov6TJJo5J2S1obEW+UrEPPDjSsiZ79AknvRMS7EfG1pIckra7weQAaVCXsiyS9P+n1aLHsW2wP2R6xPVJhWwAqqvIF3VRDhe8M0yNiWNKwxDAeaFOVnn1U0mmTXp8q6WC1cgA0pUrYd0s60/bpto+TdI2kbfWUBaBuPQ/jI+Ib2+slPStplqRNEfF6bZUBqFXPp9562hjH7EDjGrmoBsAPB2EHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSfR1ymbkc9ZZZ3Vse/PNN0vX3bBhQ2n7XXfd1VNNWdGzA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EASnGdHo5YtW9ax7ciRI6Xrjo6O1l1OapXCbvuApM8kHZb0TUQsr6MoAPWro2f/ZUR8VMPnAGgQx+xAElXDHpKes/2S7aGp3mB7yPaI7ZGK2wJQQdVh/IqIOGh7vqTttt+MiF2T3xARw5KGJcl2VNwegB5V6tkj4mDxOC5pq6QL6igKQP16DrvtObZPPPpc0uWS9tZVGIB6VRnGL5C01fbRz/lbRDxTS1WYMZYuXdqx7Ysvvihdd+vWrTVXk1vPYY+IdyWdX2MtABrEqTcgCcIOJEHYgSQIO5AEYQeS4BZXVHLuueeWtq9fv75j2/333193OShBzw4kQdiBJAg7kARhB5Ig7EAShB1IgrADSXCeHZWcffbZpe1z5szp2Pbwww/XXQ5K0LMDSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKO6N8kLcwIM/O8+OKLpe0nn3xyx7Zu98J3+6lpTC0iPNVyenYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIL72VFqyZIlpe3Lly8vbd+/f3/HNs6j91fXnt32JtvjtvdOWjbP9nbbbxePc5stE0BV0xnG3yfpimOW3SxpR0ScKWlH8RrAAOsa9ojYJenjYxavlrS5eL5Z0lX1lgWgbr0esy+IiDFJiogx2/M7vdH2kKShHrcDoCaNf0EXEcOShiVuhAHa1Oupt0O2F0pS8TheX0kAmtBr2LdJWlc8Xyfp8XrKAdCUrsN42w9KuljSSbZHJd0q6XZJj9i+XtJ7kq5uski056KLLqq0/ocfflhTJaiqa9gjYm2HpktqrgVAg7hcFkiCsANJEHYgCcIOJEHYgSS4xRWlzjvvvErrb9y4saZKUBU9O5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kwZTNyV144YWl7U899VRp+4EDB0rbV6xY0bHtq6++Kl0XvWHKZiA5wg4kQdiBJAg7kARhB5Ig7EAShB1IgvvZk7v00ktL2+fNm1fa/swzz5S2cy59cNCzA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EASnGdP7vzzzy9t7/Z7B1u2bKmzHDSoa89ue5Ptcdt7Jy27zfYHtvcUf6uaLRNAVdMZxt8n6Yoplv8lIpYWf3+vtywAdesa9ojYJenjPtQCoEFVvqBbb/vVYpg/t9ObbA/ZHrE9UmFbACrqNex3SzpD0lJJY5Lu6PTGiBiOiOURsbzHbQGoQU9hj4hDEXE4Io5IukfSBfWWBaBuPYXd9sJJL9dI2tvpvQAGQ9ffjbf9oKSLJZ0k6ZCkW4vXSyWFpAOSboiIsa4b43fj++6UU04pbd+zZ09p+yeffFLafs4553zfktCwTr8b3/WimohYO8XieytXBKCvuFwWSIKwA0kQdiAJwg4kQdiBJLjFdYa77rrrStvnz59f2v7000/XWA3aRM8OJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0lwnn2GW7x4caX1u93iih8OenYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSILz7DPclVdeWWn9J554oqZK0DZ6diAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgvPsM8DKlSs7tnWbshl5dO3ZbZ9m+3nb+2y/bntDsXye7e223y4e5zZfLoBeTWcY/42k30XEOZIulHSj7Z9JulnSjog4U9KO4jWAAdU17BExFhEvF88/k7RP0iJJqyVtLt62WdJVDdUIoAbf65jd9hJJyyS9IGlBRIxJE/8h2J5y0jDbQ5KGKtYJoKJph932CZIelXRTRHxqe1rrRcSwpOHiM6KXIgFUN61Tb7ZnayLoD0TEY8XiQ7YXFu0LJY03UyKAOnTt2T3Rhd8raV9E3DmpaZukdZJuLx4fb6RCdLVmzZqObbNmzSpd95VXXilt37VrV081YfBMZxi/QtJvJL1me0+x7BZNhPwR29dLek/S1Y1UCKAWXcMeEf+U1OkA/ZJ6ywHQFC6XBZIg7EAShB1IgrADSRB2IAlucf0BOP7440vbV61a1fNnb9mypbT98OHDPX82Bgs9O5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4k4Yj+/XgMv1TTm9mzZ5e279y5s2Pb+Hj5b4pce+21pe1ffvllaTsGT0RMeZcqPTuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJMF5dmCG4Tw7kBxhB5Ig7EAShB1IgrADSRB2IAnCDiTRNey2T7P9vO19tl+3vaFYfpvtD2zvKf56//FyAI3relGN7YWSFkbEy7ZPlPSSpKsk/VrS5xHx52lvjItqgMZ1uqhmOvOzj0kaK55/ZnufpEX1lgegad/rmN32EknLJL1QLFpv+1Xbm2zP7bDOkO0R2yPVSgVQxbSvjbd9gqSdkv4UEY/ZXiDpI0kh6Y+aGOr/tstnMIwHGtZpGD+tsNueLelJSc9GxJ1TtC+R9GREnNvlcwg70LCeb4SxbUn3Sto3OejFF3dHrZG0t2qRAJoznW/jV0r6h6TXJB0pFt8iaa2kpZoYxh+QdEPxZV7ZZ9GzAw2rNIyvC2EHmsf97EByhB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSS6/uBkzT6S9J9Jr08qlg2iQa1tUOuSqK1Xdda2uFNDX+9n/87G7ZGIWN5aASUGtbZBrUuitl71qzaG8UAShB1Iou2wD7e8/TKDWtug1iVRW6/6Ulurx+wA+qftnh1AnxB2IIlWwm77Cttv2X7H9s1t1NCJ7QO2XyumoW51frpiDr1x23snLZtne7vtt4vHKefYa6m2gZjGu2Sa8Vb3XdvTn/f9mN32LEn7JV0maVTSbklrI+KNvhbSge0DkpZHROsXYNj+haTPJf316NRatjdK+jgibi/+o5wbEb8fkNpu0/ecxruh2jpNM36dWtx3dU5/3os2evYLJL0TEe9GxNeSHpK0uoU6Bl5E7JL08TGLV0vaXDzfrIl/LH3XobaBEBFjEfFy8fwzSUenGW9135XU1RdthH2RpPcnvR7VYM33HpKes/2S7aG2i5nCgqPTbBWP81uu51hdp/Hup2OmGR+YfdfL9OdVtRH2qaamGaTzfysi4ueSfiXpxmK4ium5W9IZmpgDcEzSHW0WU0wz/qikmyLi0zZrmWyKuvqy39oI+6ik0ya9PlXSwRbqmFJEHCwexyVt1cRhxyA5dHQG3eJxvOV6/i8iDkXE4Yg4IuketbjvimnGH5X0QEQ8Vixufd9NVVe/9lsbYd8t6Uzbp9s+TtI1kra1UMd32J5TfHEi23MkXa7Bm4p6m6R1xfN1kh5vsZZvGZRpvDtNM66W913r059HRN//JK3SxDfy/5b0hzZq6FDXTyX9q/h7ve3aJD2oiWHdfzUxIrpe0k8k7ZD0dvE4b4Bqu18TU3u/qolgLWyptpWaODR8VdKe4m9V2/uupK6+7DculwWS4Ao6IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUjif1f9vw1I/2nmAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model prediction: 1\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAANr0lEQVR4nO3db6gd9Z3H8c/HbPsk7QOzuYZo3aTbSFxZWLNoXLBE19KSCGL6wNUgksXCjRJNhIVdMWAjy4Lo1n1iSLil0uzStRSi2yBhG5GwWSEUb8Q/MXdb/xDT1EtiDFglSGP87oM7kZt4z8z1zMyZk3zfL7icc+Z7zpxvp34yc87vzPwcEQJw4buo6wYADAZhB5Ig7EAShB1IgrADSfzJIN/MNl/9Ay2LCM+0vNae3fZK27+x/ZbtB+usC0C73O84u+05kn4r6buSjkh6SdKaiDhY8hr27EDL2tizL5f0VkS8ExF/lPRzSbfWWB+AFtUJ+2WSfjft8ZFi2Vlsj9oetz1e470A1FTnC7qZDhW+cJgeEWOSxiQO44Eu1dmzH5F0+bTH35D0Xr12ALSlTthfknSF7W/a/qqkOyTtbKYtAE3r+zA+Ij61fZ+kX0maI+mpiHijsc4ANKrvobe+3ozP7EDrWvlRDYDzB2EHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSQx0ymYM3ty5c0vrjz/+eGl93bp1pfX9+/eX1m+77baetXfffbf0tWgWe3YgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIJZXC9wS5YsKa1PTEzUWv9FF5XvLzZs2NCztmXLllrvjZn1msW11o9qbB+S9JGk05I+jYhr6qwPQHua+AXd30bE8QbWA6BFfGYHkqgb9pC02/Z+26MzPcH2qO1x2+M13wtADXUP46+PiPdsXyLpedv/FxF7pz8hIsYkjUl8QQd0qdaePSLeK26PSXpW0vImmgLQvL7Dbnuu7a+fuS/pe5IONNUYgGbVOYxfIOlZ22fW858R8d+NdIUvZWRkpGdt+/btA+wEw6zvsEfEO5L+qsFeALSIoTcgCcIOJEHYgSQIO5AEYQeS4FLS54Gy00QlafXq1T1ry5d3+zunFStW9KxVnR776quvltb37t1bWsfZ2LMDSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBJcSvo8cPr06dL6Z599NqBOvqhqrLxOb1VTOt9+++2l9arppC9UvS4lzZ4dSIKwA0kQdiAJwg4kQdiBJAg7kARhB5JgnH0I7Nq1q7S+atWq0nqX4+wffPBBaf3jjz/uWVu0aFHT7Zxlzpw5ra5/WDHODiRH2IEkCDuQBGEHkiDsQBKEHUiCsANJcN34AbjhhhtK60uXLi2tV42jtznOvm3bttL67t27S+sffvhhz9pNN91U+tpNmzaV1qvce++9PWtbt26tte7zUeWe3fZTto/ZPjBt2Tzbz9t+s7i9uN02AdQ1m8P4n0paec6yByW9EBFXSHqheAxgiFWGPSL2SjpxzuJbJW0v7m+XtLrZtgA0rd/P7AsiYlKSImLS9iW9nmh7VNJon+8DoCGtf0EXEWOSxiROhAG61O/Q21HbCyWpuD3WXEsA2tBv2HdKWlvcXyvpl820A6Atleez235a0o2S5ks6KumHkv5L0i8k/Zmkw5Jui4hzv8SbaV0X5GH84sWLS+v79u0rrc+fP7+0Xufa7FXXXt+xY0dp/ZFHHimtnzx5srRepup89qrtNjIyUlr/5JNPetYefvjh0tc++eSTpfVTp06V1rvU63z2ys/sEbGmR+k7tToCMFD8XBZIgrADSRB2IAnCDiRB2IEkuJR0A5YsWVJan5iYqLX+qqG3PXv29Kzdcccdpa89fvx4Xz0Nwv33319af+KJJ0rrZdut6rTgK6+8srT+9ttvl9a7xKWkgeQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJLiV9HhgfHy+t33333T1rwzyOXmXnzp2l9TvvvLO0fu211zbZznmPPTuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJME4+wBUnY9e5brrrmuok/OLPeNp2Z+r2q51tvvmzZtL63fddVff6+4Ke3YgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIJx9gbcc889pfWqa5RjZrfccktpfdmyZaX1su1e9f9J1Tj7+ahyz277KdvHbB+Ytmyz7d/bfqX4u7ndNgHUNZvD+J9KWjnD8n+LiKuLv13NtgWgaZVhj4i9kk4MoBcALarzBd19tl8rDvMv7vUk26O2x22XX0gNQKv6DftWSd+SdLWkSUk/6vXEiBiLiGsi4po+3wtAA/oKe0QcjYjTEfGZpB9LWt5sWwCa1lfYbS+c9vD7kg70ei6A4VA5zm77aUk3Sppv+4ikH0q60fbVkkLSIUnr2mtx+FWNB2c2MjLSs3bVVVeVvvahhx5qup3Pvf/++6X1U6dOtfbeXakMe0SsmWHxT1roBUCL+LkskARhB5Ig7EAShB1IgrADSXCKK1q1adOmnrX169e3+t6HDh3qWVu7dm3paw8fPtxwN91jzw4kQdiBJAg7kARhB5Ig7EAShB1IgrADSTDOjlp27Sq/1ujSpUsH1MkXHTx4sGftxRdfHGAnw4E9O5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kwTh7A2yX1i+6qN6/qatWrer7tWNjY6X1Sy+9tO91S9X/27qcrppLfJ+NPTuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJME4ewO2bt1aWn/sscdqrf+5554rrdcZy257HLzN9W/btq21dV+IKvfsti+3vcf2hO03bG8sls+z/bztN4vbi9tvF0C/ZnMY/6mkf4iIv5D0N5LW275K0oOSXoiIKyS9UDwGMKQqwx4RkxHxcnH/I0kTki6TdKuk7cXTtkta3VKPABrwpT6z214saZmkX0taEBGT0tQ/CLYv6fGaUUmjNfsEUNOsw277a5J2SHogIv5QdfLHGRExJmmsWEf00ySA+mY19Gb7K5oK+s8i4pli8VHbC4v6QknH2mkRQBMcUb6z9dQufLukExHxwLTlj0v6ICIetf2gpHkR8Y8V67og9+yLFi0qre/bt6+0PjIyUlof5tNIq3o7evRoz9rExETpa0dHyz/9TU5OltZPnjxZWr9QRcSMh92zOYy/XtJdkl63/Uqx7CFJj0r6he0fSDos6bYG+gTQksqwR8SLknp9QP9Os+0AaAs/lwWSIOxAEoQdSIKwA0kQdiCJynH2Rt/sAh1nr7JixYrS+urVq0vrGzduLK0P8zj7hg0beta2bNnSdDtQ73F29uxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kATj7OeBlStXltbLzvuumrZ4586dpfWqKZ+rrlh08ODBnrXDhw+Xvhb9YZwdSI6wA0kQdiAJwg4kQdiBJAg7kARhB5JgnB24wDDODiRH2IEkCDuQBGEHkiDsQBKEHUiCsANJVIbd9uW299iesP2G7Y3F8s22f2/7leLv5vbbBdCvyh/V2F4oaWFEvGz765L2S1ot6e8kfRwR/zrrN+NHNUDrev2oZjbzs09Kmizuf2R7QtJlzbYHoG1f6jO77cWSlkn6dbHoPtuv2X7K9sU9XjNqe9z2eL1WAdQx69/G2/6apP+R9C8R8YztBZKOSwpJ/6ypQ/27K9bBYTzQsl6H8bMKu+2vSHpO0q8i4okZ6oslPRcRf1mxHsIOtKzvE2E8dfnQn0iamB704ou7M74v6UDdJgG0Zzbfxn9b0v9Kel3SmbmBH5K0RtLVmjqMPyRpXfFlXtm62LMDLat1GN8Uwg60j/PZgeQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSVRecLJhxyW9O+3x/GLZMBrW3oa1L4ne+tVkb4t6FQZ6PvsX3twej4hrOmugxLD2Nqx9SfTWr0H1xmE8kARhB5LoOuxjHb9/mWHtbVj7kuitXwPprdPP7AAGp+s9O4ABIexAEp2E3fZK27+x/ZbtB7vooRfbh2y/XkxD3en8dMUcesdsH5i2bJ7t522/WdzOOMdeR70NxTTeJdOMd7rtup7+fOCf2W3PkfRbSd+VdETSS5LWRMTBgTbSg+1Dkq6JiM5/gGF7haSPJf37mam1bD8m6UREPFr8Q3lxRPzTkPS2WV9yGu+Weus1zfjfq8Nt1+T05/3oYs++XNJbEfFORPxR0s8l3dpBH0MvIvZKOnHO4lslbS/ub9fUfywD16O3oRARkxHxcnH/I0lnphnvdNuV9DUQXYT9Mkm/m/b4iIZrvveQtNv2ftujXTczgwVnptkqbi/puJ9zVU7jPUjnTDM+NNuun+nP6+oi7DNNTTNM43/XR8RfS1olaX1xuIrZ2SrpW5qaA3BS0o+6bKaYZnyHpAci4g9d9jLdDH0NZLt1EfYjki6f9vgbkt7roI8ZRcR7xe0xSc9q6mPHMDl6Zgbd4vZYx/18LiKORsTpiPhM0o/V4bYrphnfIelnEfFMsbjzbTdTX4Pabl2E/SVJV9j+pu2vSrpD0s4O+vgC23OLL05ke66k72n4pqLeKWltcX+tpF922MtZhmUa717TjKvjbdf59OcRMfA/STdr6hv5tyVt6qKHHn39uaRXi783uu5N0tOaOqw7pakjoh9I+lNJL0h6s7idN0S9/YempvZ+TVPBWthRb9/W1EfD1yS9Uvzd3PW2K+lrINuNn8sCSfALOiAJwg4kQdiBJAg7kARhB5Ig7EAShB1I4v8B/55jyAhO1i4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model prediction: 0\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAANTElEQVR4nO3dYahc9ZnH8d9vTRsxDZK7IdlLGjY1Cq4EN9UgimFVSmM2IrGoS0JYUpXevqjQYl9UVKioCyLbLPvGwC1K06WbUjRiqKWthLiub0puJNVr77bGkDZpLokxhCYSqOY+++KeyDXeOXMzc86cuXm+H7jMzHnmnPNw9JdzZv4z83dECMDF72+abgBAbxB2IAnCDiRB2IEkCDuQxJxe7sw2b/0DNYsIT7e8qzO77bW2f297v+2Hu9kWgHq503F225dI+oOkr0o6LGmPpI0R8buSdTizAzWr48x+g6T9EXEgIv4q6aeS1nexPQA16ibsSyQdmvL4cLHsU2wP2R6xPdLFvgB0qZs36Ka7VPjMZXpEDEsalriMB5rUzZn9sKSlUx5/UdKR7toBUJduwr5H0lW2v2T785I2SNpZTVsAqtbxZXxEfGz7QUm/knSJpOcj4p3KOgNQqY6H3jraGa/ZgdrV8qEaALMHYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0l0PGUzeue6664rre/YsaNlbdmyZRV30z/WrFlTWh8bG2tZO3ToUNXt9L2uwm77oKRTks5K+jgiVlXRFIDqVXFmvy0ijlewHQA14jU7kES3YQ9Jv7a91/bQdE+wPWR7xPZIl/sC0IVuL+NvjogjthdJetX2/0XE61OfEBHDkoYlyXZ0uT8AHerqzB4RR4rbY5JeknRDFU0BqF7HYbc9z/b8c/clrZE0WlVjAKrVzWX8Ykkv2T63nf+OiF9W0hU+5fbbby+tz507t0ed9Jc777yztH7//fe3rG3YsKHqdvpex2GPiAOS/rHCXgDUiKE3IAnCDiRB2IEkCDuQBGEHkuArrn1gzpzy/wzr1q3rUSezy969e0vrDz30UMvavHnzStf98MMPO+qpn3FmB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkGGfvA7fddltp/aabbiqtP/PMM1W2M2ssWLCgtH7NNde0rF122WWl6zLODmDWIuxAEoQdSIKwA0kQdiAJwg4kQdiBJBzRu0lass4Is2LFitL6a6+9Vlr/4IMPSuvXX399y9rp06dL153N2h231atXt6wNDg6Wrvv+++930lJfiAhPt5wzO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kwffZe+Cxxx4rrbf7DfO1a9eW1i/WsfSBgYHS+i233FJan5iYqLKdWa/tmd3287aP2R6dsmzA9qu23y1uy39FAEDjZnIZ/yNJ559aHpa0KyKukrSreAygj7UNe0S8LunEeYvXS9pW3N8m6a5q2wJQtU5fsy+OiHFJiohx24taPdH2kKShDvcDoCK1v0EXEcOShqW8X4QB+kGnQ29HbQ9KUnF7rLqWANSh07DvlLS5uL9Z0svVtAOgLm0v421vl3SrpIW2D0v6vqSnJf3M9gOS/iTp3jqb7Hf33HNPab3d/Or79+8vrY+MjFxwTxeDRx99tLTebhy97PvuJ0+e7KCj2a1t2CNiY4vSVyruBUCN+LgskARhB5Ig7EAShB1IgrADSfAV1wrce2/5yGO76YGfffbZKtuZNZYtW1Za37RpU2n97NmzpfWnnnqqZe2jjz4qXfdixJkdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5JgnH2GLr/88pa1G2+8sattb926tav1Z6uhofJfK1u4cGFpfWxsrLS+e/fuC+7pYsaZHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSYJx9hubOnduytmTJktJ1t2/fXnU7F4Xly5d3tf7o6Gj7J+ETnNmBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnG2Wfo1KlTLWv79u0rXffaa68trQ8MDJTWT5w4UVrvZ4sWLWpZazfVdTtvvPFGV+tn0/bMbvt528dsj05Z9rjtP9veV/yVT0AOoHEzuYz/kaS10yz/j4hYWfz9otq2AFStbdgj4nVJs/c6EoCk7t6ge9D2W8Vl/oJWT7I9ZHvE9kgX+wLQpU7DvlXSckkrJY1L+kGrJ0bEcESsiohVHe4LQAU6CntEHI2IsxExIemHkm6oti0AVeso7LYHpzz8miS+awj0ubbj7La3S7pV0kLbhyV9X9KttldKCkkHJX2zvhb7w5kzZ1rW3nvvvdJ177777tL6K6+8UlrfsmVLab1OK1asKK1fccUVpfWyOdgjopOWPjExMdHV+tm0DXtEbJxm8XM19AKgRnxcFkiCsANJEHYgCcIOJEHYgSTc7fDHBe3M7t3Oeujqq68urT/xxBOl9TvuuKO0XvYz1nU7fvx4ab3d/z9l0y7b7qinc+bPn19aLxsuvZhFxLQHljM7kARhB5Ig7EAShB1IgrADSRB2IAnCDiTBOHsfWLlyZWn9yiuv7E0j03jhhRe6Wn/btm0ta5s2bepq23Pm8Evo02GcHUiOsANJEHYgCcIOJEHYgSQIO5AEYQeSYKCyD7Sb8rldvZ8dOHCgtm23+5nr0VGmM5iKMzuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJME4O2pV9tvw3f5uPOPoF6btmd32Utu7bY/Zfsf2t4vlA7Zftf1ucbug/nYBdGoml/EfS/puRPyDpBslfcv2NZIelrQrIq6StKt4DKBPtQ17RIxHxJvF/VOSxiQtkbRe0rnfHNom6a6aegRQgQt6zW57maQvS/qNpMURMS5N/oNge1GLdYYkDXXZJ4AuzTjstr8g6UVJ34mIv8z0zZWIGJY0XGyDH5wEGjKjoTfbn9Nk0H8SETuKxUdtDxb1QUnH6mkRQBVm8m68JT0naSwitkwp7ZS0ubi/WdLL1beH2S4iavvDhZnJZfzNkv5V0tu29xXLHpH0tKSf2X5A0p8k3VtLhwAq0TbsEfGGpFYv0L9SbTsA6sLHZYEkCDuQBGEHkiDsQBKEHUiCr7iiVpdeemnH6545c6bCTsCZHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSYJwdtbrvvvta1k6ePFm67pNPPllxN7lxZgeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJBhnR6327NnTsrZly5aWNUnavXt31e2kxpkdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Jwu3mubS+V9GNJfydpQtJwRPyn7cclfUPS+8VTH4mIX7TZFpNqAzWLiGlnXZ5J2AclDUbEm7bnS9or6S5J/yLpdET8+0ybIOxA/VqFfSbzs49LGi/un7I9JmlJte0BqNsFvWa3vUzSlyX9plj0oO23bD9ve0GLdYZsj9ge6a5VAN1oexn/yRPtL0j6H0n/FhE7bC+WdFxSSHpSk5f697fZBpfxQM06fs0uSbY/J+nnkn4VEZ/59kJxxv95RKxosx3CDtSsVdjbXsbbtqTnJI1NDXrxxt05X5M02m2TAOozk3fjV0v6X0lva3LoTZIekbRR0kpNXsYflPTN4s28sm1xZgdq1tVlfFUIO1C/ji/jAVwcCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0n0esrm45L+OOXxwmJZP+rX3vq1L4neOlVlb3/fqtDT77N/Zuf2SESsaqyBEv3aW7/2JdFbp3rVG5fxQBKEHUii6bAPN7z/Mv3aW7/2JdFbp3rSW6Ov2QH0TtNndgA9QtiBJBoJu+21tn9ve7/th5vooRXbB22/bXtf0/PTFXPoHbM9OmXZgO1Xbb9b3E47x15DvT1u+8/Fsdtne11DvS21vdv2mO13bH+7WN7osSvpqyfHreev2W1fIukPkr4q6bCkPZI2RsTvetpIC7YPSloVEY1/AMP2P0k6LenH56bWsv2MpBMR8XTxD+WCiPhen/T2uC5wGu+aems1zfjX1eCxq3L68040cWa/QdL+iDgQEX+V9FNJ6xvoo+9FxOuSTpy3eL2kbcX9bZr8n6XnWvTWFyJiPCLeLO6fknRumvFGj11JXz3RRNiXSDo05fFh9dd87yHp17b32h5quplpLD43zVZxu6jhfs7XdhrvXjpvmvG+OXadTH/erSbCPt3UNP00/ndzRFwn6Z8lfau4XMXMbJW0XJNzAI5L+kGTzRTTjL8o6TsR8Zcme5lqmr56ctyaCPthSUunPP6ipCMN9DGtiDhS3B6T9JImX3b0k6PnZtAtbo813M8nIuJoRJyNiAlJP1SDx66YZvxFST+JiB3F4saP3XR99eq4NRH2PZKusv0l25+XtEHSzgb6+Azb84o3TmR7nqQ16r+pqHdK2lzc3yzp5QZ7+ZR+mca71TTjavjYNT79eUT0/E/SOk2+I/+epEeb6KFFX1dI+m3x907TvUnarsnLuo80eUX0gKS/lbRL0rvF7UAf9fZfmpza+y1NBmuwod5Wa/Kl4VuS9hV/65o+diV99eS48XFZIAk+QQckQdiBJAg7kARhB5Ig7EAShB1IgrADSfw/gg1DYNhx/+sAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model prediction: 4\n"
     ]
    }
   ],
   "source": [
    "# Predict 5 images from validation set.\n",
    "n_images = 5\n",
    "test_images = x_test[:n_images]\n",
    "predictions = conv_net(test_images)\n",
    "\n",
    "# Display image and model prediction.\n",
    "for i in range(n_images):\n",
    "    plt.imshow(np.reshape(test_images[i], [28, 28]), cmap='gray')\n",
    "    plt.show()\n",
    "    print(\"Model prediction: %i\" % np.argmax(predictions.numpy()[i]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
